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Undergraduate students’ written exams were analyzed from a freshman-level 
mathematics course that emphasized, among other topics, the study of mathematical 
logic. Findings indicate that on questions related to the negation of a conditional 
sentence, students performed much better when given natural-language contexts than 
they did on questions presented in abstract form. However, students showed 
improvement over time only on the abstract questions. These results raise doubts about 
the validity and efficacy of using natural-language examples to promote and test the 
understanding of symbolic logic. 

INTRODUCTION 

This report describes significant findings about learning and teaching logic. The data are 
from a larger study of the nature of skills and understandings developed in an 
undergraduate course entitled "The Language of Mathematics." Among other 
mathematical topics, the course emphasized logical connectives, truth tables, and the 
abstract logical forms of mathematical and non-mathematical sentences. Findings 
indicate that, when quiz and test questions attempted to measure the students’ 
understanding of logic, the context of the questions was very significant. A substantial 
difference in students’ responses was identified between problems presented in natural- 
language contexts and parallel problems in abstract mathematical contexts. These results 
raise doubts about the validity and efficacy of using natural-language examples to 
promote and test the understanding of symbolic logic. 

Many texts use natural-language examples from familiar contexts to illustrate results 
from symbolic logic. The form "H => C" (Hy pothesis implies Conclusion) can be 
discussed by considering the example, "If you liked the book, you will like the movie," 
and asking what can be deduced from "You did not like the book."II(Bennet and Briggs, 
2002). However, when examples and questions are phrased in natural language, 
students’ prior knowledge may interfere with, and perhaps even eliminate, the intended 
instructional focus on logical form (Gregg, 1997). It has long been recognized that 
humans have an understanding of conditionals in a social context that may not be closely 
related to their understanding of conditionals in a more-abstract context such as 
mathematics (Devlin, 2000). 

Judging by the overwhelming dominance of natural-language examples of logic in 
“liberal arts” mathematics texts, one might suppose it has been demonstrated that analysis 
of natural-language examples illuminates the logical forms themselves and is an effective 
way to teach symbolic logic. In spite of the fact that logic technically addresses the 
abstract forms of sentences and not their meanings, our findings suggest that responses to 
natural-language examples are strongly confounded by the students' prior knowledge of 
the context and that these answers may provide minimal information about the students’ 
grasp of the abstract logical forms supposedly being considered. 
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The findings of other research studies of students’ use of deductive reasoning also 
indicate that certain characteristics of the tasks, other than their logical structure, 
influence the students’ responses. Gregg (1997) describes a high school classroom 
exchange in which students appeared to assess the truth of a sequence of statements 
dealing with a “real life” situation on the basis of personal opinion and experience. 
Kiicherman & Hoyles (2002) found that many 13 year-old students preferred to reason 
through particular examples rather than on the basis of logical structure when given tasks 
based on familiar number theory relationships. The presentation of information as a 
sequenced set of statements may also affect responses. Ferrari & Giraudi (2001), 
working with 10 th graders, and Barkai et al, (2002), in a study of elementary school 
teachers, documented that the subjects tended to regard each of the statements in the set 
as an isolated topic rather than as being logically related to each other. 

The study reported here adds to this body of knowledge by investigating how students 
responded to logic statements presented in different symbol/language systems, and with 
varying degrees of contextural familiarity. In addition, the study examines the often 
unexpected nature of information obtained from classroom assessment. 



THE STUDY 

The sub-set of findings that are of interest here were taken from the students’ work 
during the second half of the semester. The emphasis during this time was on the study 
of logic for mathematics, including logical connectives and equivalents, and ways of 
expressing mathematical generalizations and existence statements. Examples of logical 
statements and expressions were presented in pure logical form, algebraic language, 
natural language, and through truth tables and Venn diagrams. 

The “Language of Mathematics course was offered at the freshman level to 35 students. 
Six were elementary education majors pursuing a mathematics option. Twelve, who 
were enrolled in health-related fields, were fulfilling a prerequisite for an introductory- 
level statistics course. The remaining sixteen students were taking the course to complete 
a minimum math core-curriculum requirement for graduation. 

The data for the larger research study consisted of the students’ written responses to all 
the quizzes, the four chapter tests, and the final exam. A small subset of the data that was 
found to be particularly striking is analyzed here. It consists of four test questions that 
pertain to the negation of conditional sentences. These problems each present a sentence 
with the form "H => C" and request its negation. 

The four questions are presented below. The date on which each test was administered 
and the instructional focus prior to that exam are noted. Examples of common student 
responses, their underlying logical form, and the percentage of students giving that type 
of answer are also provided. 



A) Exam 3, April 5 (after truth-table logic) Simplify: not [{not A) => B] 
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11.4% 



22.9% 

14.3% 

48.6% 



Correct answer: (not A) and (not B). [H and not C] 

Typical incorrect answers: 

A and (not B) [not H and not C], A and B [not H and C]. 

A or not B [not H or not C ], A or B [not H or C], not A or B [H or C]. 

A => not B [not H => not C], A =>B [not H => C]. 

B) Exam 3, April 5: How can this law be broken? “Don’t drive if you’re 

Correct answers: 

71 . 4 % 

You're intoxicated and you drive. [H and not C ] 

34 . 3 % 

Driving while/when intoxicated. [not C and H] 

31 . 4 % 

Driving intoxicated. [not C and H] 5.7% 

Typical incorrect answers: 

You cannot drive and be intoxicated, [not (not C and H)] 2.9% 

Drive if you're intoxicated. [H => not C] 17.1% 

Don't drive if you're not intoxicated, [not H => C] 5.7% 

Not H would be drive [defined negation ofC] 2.9% 

C ) Exam 4, April 26 (after studying quantifiers) 

Negate and put in positive form: “All who wander are lost.” 

(Students knew that “positive form” requested that the “not” be eliminated or, at 
least, moved inside the statement as far as conveniently possible.) 

Correct answers: [[3 x st. H and not C] 31.3% 

There exists a wanderer who is not lost. 28.1% 

There is one who wanders and knows where he is going. 3.1% 

Less than fully correct answers: 28.1% 

Not all who wander are lost, [not (H => C): not simplified as 
requested .] 21.8% 

You wander and are not lost. [H and not C: no quantifier given .] 3.1% 

There exists one that is not lost. [3 x such that x G not C] 3.1% 

Incorrect answers: 4 0.6% 

There exist people who are lost but do not wander. [3 x s.t. C and not H] 3.1% 

All who wander are not lost. [H => not C] 28.1% 

Not all who wander are not lost, [not ( H => not C)] 6.2% 

Not all who are lost wander, [not (C => H)\ 3.1% 

D ) Final Exam, May 9: Here’s a sentence: If x E S, then f (x) = 0. What is its 

negation? 

Correct answer: 3x such that x G S and f(x) ^ 0. [3 x st. H and not C] 5.9% 

Less than fully correct answer: x G S and f(x) ^ 0. [H and not C] 29.4% 

Incorrect answers: 

If x G S, then f(x) * 0. [H => not C] 38.2% 

If x (£ S, then f(x) ^ 0. [not H => not C] 14.7% 

If x f S, then f(x) = 0. [not H => C] 2.9% 
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Other answers 8.8% 

ANALYSIS 

For each question, each student’s response was examined to determine its underlying 
logical form. This sorting revealed six general Answer Categories. 

• (6) mathematically correct, 

• (5) “close” response having only a minor error, 

• (4) incorrect response that included the connective “and,” 

• (3) incorrect response that included the connective “or,” 

• (2) “if, . . . then” statement, 

• (1) incorrect response that did not use a logical connective. 

The numerical designations for the six categories reflect a subjective measure of the 
mathematical “correctness” of a student’s response. The “close” category is only 
relevant for the last two problems (C and D ) because these questions are mathematical 
generalizations, and their negations must be stated as existence statements. For these 
questions, student responses were categorized as “close” if they were in the form “H and 
not C” but were not written as existence statements. A response for question C was also 
considered ’’close” if it was written in the form “not (H => C).” That is, the negation 
operation was stated but not carried out. Table 1 shows the percentage of responses 
classified by Answer Category for each question. 



Answer Category -»■ 


6 


5 


4 


3 


2 


1 


Abstract: A 


11 


0 


23 


14 


49 


3 


Natural Language: B 


71 


0 


3 


3 


23 


0 


Natural Language: C 


31 


28 


3 


0 


38 


0 


Abstract: D 


6 


29 


0 


0 


56 


9 



Table 1. Percentage distribution of Answer Categories for each question 



FINDINGS 

Two research questions were addressed. 1) Does the distribution of student responses 
vary with the form and content of the question? 2) How does the distribution of 
responses vary over time? In other words, did some types of questions present more 
difficulty to students, and was there improvement in their measured knowledge as the 
course progressed? In order to address these questions, the distributions of the different 
types of student responses were compared across problems, and differences in the form 
and content of the four problems were characterized. 

Distribution of Student Responses 

At the level of the entire class of students, a comparison of the distribution of responses 
shown in Table 1 reveals a “quick” answer to the two research questions. More students 
did well on the two natural-language questions than on the other two problems. For the 
natural-language problems B and C, over half of the students wrote correct or nearly 
correct answers (71% and 59% respectively). In contrast, for the more abstract questions 
A and D, only 11% and 35% wrote correct or nearly correct answers. 

Overall, the improvement in the percentage of correct and “close” (good) responses over 
time was a function of the type of question being asked. Responses to the two natural- 
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language questions fell from 71% (B) to 59% (C) between Exam 3 and Exam 4. On the 
other hand, the percentage of good answers to the two abstract questions improved from 
11% (A) on Exam 3 to 35% ( D ) on the Final Exam. The fact that the students improved 
on the abstract questions but not on the natural-language questions suggests that their 
responses to the natural-language questions continued to be based throughout the course 
on contextual sources of information. The natural language contexts appeared to 
overwhelm any gains that the students made in their study of logic. 



Characterization of Problem Form and Content 

A comparison of the four questions examined five aspects of problem form and content; 
1) the symbol system used to express the logical statement, 2) the context of the 
statement, 3) the level of abstraction of the objects symbolized, 4) whether the underlying 
logical form was implicit or explicitly given in the statement, and 5) the complexity of 
the problem related to a straightforward “translation” of the logical form into that 
particular context. These aspects are summarized in Table 2. The total percentage for the 
correct and close answers is shown for each question. 



Formal. Abstract Symbol Systems 
A: Simplify: not [(not A) => B] (11%) 

Symbol system: Mathematical logic 
Context: Logical objects 
Level of abstraction: High - Objects are 

abstracted as letters, operations as 
symbols 

Form: Explicitly stated 

Complexity: Moderate - the hypothesis is 

given as “not A” 

D\ Here’s a sentence: Ifx E S, thenf(x) = 0. 

What is its negation? (35%) 

Symbol System: Algebra 
Context: Sets and functions 
Level of abstraction: Semi-abstract - but 

algebraic objects are generalizations 
Form: Explicitly given by the structure 

“if. ..then” - implicit generalization 
Complexity: Moderate -objects in the 

hypothesis and conclusion are abstract 
generalizations 



Natural Language. Experience-Based 
B\ How can this law be broken? (77%) 

“ Don 7 drink if you ’re intoxicated. ” 
Symbol system: Natural language 
Context: Rules of driving 
Level of abstract: Low - experiential 

world 

Form: Somewhat implicit - “If’ present in 
statement 

Complexity : Low /context - rules are 

familiar phrases. Moderate / form — 
hypothesis, conclusion in reverse order 

C : Negate and put in positive form : 

“All who wander are lost. ” (59%) 
Symbol system: Natural language 
Context: Adage-like statement 
Level of abstract: Low - experiential 

world 

Form: Implicit logical form, generalization 
explicitly expressed by “all” 
Complexity: Moderate - negation difficult 
to express in fluent English 



Table 2. Form and content analysis 



It can be seen that, as a class, the students tended to do less well on the two questions that 
were expressed in a formal, abstract symbol system, even though in these questions, the 
logical form under examination was explicitly available to the students. It is interesting 
to note that the most difficult question (A) and the easiest one ( B ) were sequential 
problems on the same exam. These two questions epitomize the key factors that 
appeared to affect problem accessibility - abstract logical form and contextual 
familiarity. 
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Negation 

The use of abstract patterns or familiar context also factors into the mechanics of 
generating an answer to a given question. The ability to produce the correct response 
consists of two parts - knowing how to represent the negation of either the given 
hypothesis or conclusion, and also knowing the correct logical form to use with this 
negation. An examination of the students’ responses shows that the symbol system in 
which the conditional statement was expressed influenced the students’ abilities to 
successfully create an appropriate negation. 

Negating the component expressions in the conditional statement appeared easier for the 
students to express in the abstract symbol systems than in the natural-language 
statements. These differences may be the result of the lack of ambiguity inherent in the 
abstract systems, as compared to the variety of English phrases that can be used to 
express a single thought. 

In the two abstract symbol systems (logical form and algebra), students negated objects 
by simply placing “not” in front of a letter or by drawing a slanted line through the 
symbols “E” and “=”. In contrast, negating several of the natural-language phrases 
appeared to present problems for some students. In question B it was much easier to talk 
about “driving intoxicated” than it was in C to describe wandering while “not lost.” In B, 
only one of the students (2.9%) stated the negation but did not carry it out, writing a 
response in the form “not (not C and H)”. In contrast, ten of the students (31.1%) wrote 
similar forms for question C; “not (H => C),” “not (H => not C),” and “not (C => H)”. 
None of the students gave such responses for the two abstract questions. 

CONCLUSIONS 

There is not enough evidence in the data analyzed for this particular study to draw firm 
conclusions. Our findings do support Devlin’s (p. 118, 2000) claim that “people reason 
much better about familiar, everyday objects and circumstances than they do about 
abstract objects in unfamiliar settings, even if the logical structure of the task is the 
same.” In addition, the potential impact of our findings on the way that mathematical 
logic is taught are sufficiently significant that further study is called for. Our results 
suggest the following interpretations. 

One way to make sense of the variations in the students’ responses is to consider the kind 
of knowledge that the students might have been applying as they answered each question. 
From this perspective, it can be conjectured that, for many students, the responses to the 
abstract questions A and D were based on their ability to memorize or learn a particular 
abstract pattern of logical symbols, while the answers to the natural-language questions B 
and C were created from the students’ knowledge of experiential relationships. The wide 
variation in the answers to question A, which was expressed in pure logical form, 
suggests that many students had not yet mastered the appropriate pattern for this form. 
Yet, these same students were able, on the very next test question ( B ), to produce a 
logically correct statement through their knowledge of the rules of driving. We suggest 
that these differences in responses are linked to the way that the conditional statement is 
expressed either by abstract patterns or within a familiar context. 
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It appears that the study of the forms of mathematical logic during the second half of the 
course had an impact on the students’ ability to correctly respond to abstractly stated 
conditional statements, but not to such statements when they were written in a natural- 
language context. As was noted, aspects of the natural language may have had more 
impact on the students’ inability to produce a correct response than did any lack of 
understanding of logical form. 

At issue, is whether and to what degree the four problems can be considered to represent 
examples of mathematical logic. Does a knowledge of the forms of mathematical logic 
reside only within symbolic systems, or can its applications be found in everyday 
experiences expressed in natural language? The percentage distribution of Answer 
Categories across the four problems and the form and content analysis of each question 
suggest instructional and assessment challenges. It appears that different kinds of 
knowledge are being tested across the set of problems and that this knowledge is related 
to the nature of the symbol system or context in which each question is posed. 

Gregg (1997) wrestled with this question in his analysis of a classroom episode from a 
tenth grade geometry class. Even when informed that the statement, If a student likes 
geometry, then he or she will pass, was assumed to be true, the students assessed the truth 
of the resulting contrapositive, If a student doesn’t pass, then he or she does not like 
geometry, on the basis of personal opinion rather than on logical equivalence. To the 
students, the statement was meaningful on what Gregg termed a “semantic level.” Gregg 
(p. 547) concluded that the use of “real-life” examples may “actually contribute to 
difficulties in the traditional approach to teaching conditionals and the notion of logical 
equivalence.” 

In this report, the detailed examination of the students’ responses, in conjunction with the 
form and content analysis of the four statements, highlights the complex task of 
developing appropriate concept images for topics of mathematical logic. There is more 
to this task than simply mastering a set of truth-table definitions or sequences of logical 
patterns. Logical form needs to be encountered in many contexts, with explicit links 
made between the form and the, sometimes overriding, focus of a given context. In 
addition, care must be taken in the kinds of inferences that are drawn from particular 
types of assessment questions. It is important for teachers to be aware of the alternative 
sources of knowledge that students may bring to bear when confronted with particular 
types of examples. 
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